{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python 机器学习实战 ——代码样例\n",
    "\n",
    "# 第五章 列表生成式"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1.\t使用列表生成式代替循环语句\n",
    "\n",
    "下面举一个简单的循环处理的例子，类似于一般的教科书上的写法，功能是对一个列表中的每个元素计算长度，并将长度添加在元素内容后面，生成一个新的列表。用循环语句来写当然无可厚非。学习任何编程语言，我们都会去学习如何处理判断、循环等这些基本的编程理念。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['a1', 'bb2', 'ccc3', 'a12']\n"
     ]
    }
   ],
   "source": [
    "# 计算列表中的每个元素长度，然后将长度添加在元素内容后面，生成一个新的列表。\n",
    "\n",
    "list_test = ['a', 'bb', 'ccc', 'a1']\n",
    "list_new = []\n",
    "\n",
    "for i in list_test:\n",
    "    new_item = i + str(len(i))\n",
    "    list_new.append(new_item)\n",
    "\n",
    "print(list_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面这个例子非常简单，也是 Python 程序中经常会出现的代码片段，属于平铺直叙的写法。然而我们推荐用列表生成式的方式来简化程序。列表生成式的好处就是自带初始化，按照条件生成列表中的元素，完成上述功能的代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['a1', 'bb2', 'ccc3', 'a12']\n"
     ]
    }
   ],
   "source": [
    "# 计算列表中的每个元素长度，然后将长度添加在元素内容后面，生成一个新的列表；\n",
    "# 使用列表生成式\n",
    "\n",
    "list_test = ['a', 'bb', 'ccc', 'a1']\n",
    "list_new = [i + str(len(i)) for i in list_test]\n",
    "print(list_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到程序代码简洁了很多，列表生成式用一行代码来代替了四行代码，节约的就是列表初始化、for 循环说明、列表元素追加这些内容。鉴于 Python 中数据结构 list 和循环是最常见的，每一个地方节约几行代码，总量不容小觑，并且在代码的可读性方面几乎没有损失。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2.\t列表生成式的概念\n",
    "\n",
    "Python 中有不少概念有时比较费解，毕竟这是一门发展了几十年的语言，比如至今还有人在为了使用 Python 2 还是 Python 3 乐此不疲的争论。\n",
    "\n",
    "List Comprehension 是一个编程语言中的概念，在 wiki 上有专门的解释 ：\n",
    "\n",
    "“List comprehension is a syntactic construct available in some programming languages for creating a list based on existing lists. It follows the form of the mathematical set-builder notation (set comprehension.) as distinct from the use of map and filter functions.”\n",
    "\n",
    "“列表生成式是一些编程语言中的语法结构，可以根据现有的列表创建新列表。它遵循了数学中“集合生成”的概念，并与 map 和 filter 函数的使用不同。”\n",
    "\n",
    "List Comprehension 在 Python 中的实现主要就是列表生成式。在 wiki 的 List Comprehension 的 Python 的词条里面解释到，Python 使用下面的语法来对一个有限定的列表来进行新的列表的生成：\n",
    "\n",
    "s = [2x for x in range(100) if x*2 > 3]\n",
    "\n",
    "在 Python 的官方文档中是这样解释的 ：\n",
    "\n",
    "“List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition.”\n",
    "\n",
    "“列表生成式为创建列表提供了一种简洁的方式。常规的创建列表的方法是对另一个序列或可迭代对象中的每一个元素进行一些操作，然后生成新的元素，或者在满足某个特定条件的情况下创建这些元素的子集。”\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['h', 'e', 'l', 'l', 'o', 'w', 'o', 'r', 'l', 'd']\n",
      "<class 'list'>\n"
     ]
    }
   ],
   "source": [
    "s = 'hello world'\n",
    "comp = [x for x in s if x !=' ']\n",
    "print(comp)\n",
    "print(type(comp))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['我', '们', '来', '测', '试', '一', '下', '中', '文']\n",
      "<class 'list'>\n"
     ]
    }
   ],
   "source": [
    "s = '我们来测试一下中文'\n",
    "comp = [x for x in s]\n",
    "print(comp)\n",
    "print(type(comp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 5.3.\t字典和集合的生成式\n",
    "\n",
    "实际上，列表生成式这个概念在 Python 中被泛化了。用刚才这样的语法，不但可以生成列表，还可以生成字典 dict 和集合 set。我们看看下面这个例子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'e', 'w', 'h', ' ', 'l', 'r', 'o', 'd'}\n",
      "<class 'set'>\n"
     ]
    }
   ],
   "source": [
    "s = 'hello world'\n",
    "comp = {x for x in s}\n",
    "print(comp)\n",
    "print(type(comp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果我们要用了列表生成式来写一个生成”字典“的话，可以用 key:value 这样的方式，这也是 python 强大的地方，你觉得应该是怎么样比较自然的，python 语法的设计就会是这样比较人性化的。\n",
    "\n",
    "严格来说，字典生成式是这样的语法：{key: value for (key, value) in iterable}\n",
    "\n",
    "zip() 函数用于将可迭代的对象作为参数，将对象中对应的元素打包成一个个元组，然后返回由这些元组组成的列表。\n",
    "\n",
    "这里需要一个生成连续的 key 值的列表，所以用了 zip() 函数。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 'h', 1: 'e', 2: 'l', 3: 'l', 4: 'o', 5: ' ', 6: 'w', 7: 'o', 8: 'r', 9: 'l', 10: 'd'}\n",
      "<class 'dict'>\n"
     ]
    }
   ],
   "source": [
    "s = 'hello world'\n",
    "dict_comp = {k:v for (k,v) in zip(range(11),s)}\n",
    "print(dict_comp)\n",
    "print(type(dict_comp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果对 zip() 函数不清楚的话，可以看一下下面的执行效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 h\n",
      "1 e\n",
      "2 l\n",
      "3 l\n",
      "4 o\n",
      "5  \n",
      "6 w\n",
      "7 o\n",
      "8 r\n"
     ]
    }
   ],
   "source": [
    "s = 'hello world'\n",
    "for k,v in zip(range(9),s):\n",
    "    print(k,v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们还可以把程序稍微优化一下，业务逻辑分离，因为 range(9) 这样的写法是不合适的，我们可以继续优雅的使用 python 的连续变量赋值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 'h', 1: 'e', 2: 'l', 3: 'l', 4: 'o', 5: ' ', 6: 'w', 7: 'o', 8: 'r', 9: 'l', 10: 'd'}\n",
      "<class 'dict'>\n"
     ]
    }
   ],
   "source": [
    "s, i = 'hello world', len(s)\n",
    "dict_comp = {k:v for (k,v) in zip(range(i),s)}\n",
    "print(dict_comp)\n",
    "print(type(dict_comp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这个例子过于简单，实际上不用字典生成式，直接用 dict() 来进行转换也是可以的，但是如果在转换 key 和 value 的时候有一些逻辑的话，字典生成式还是一个不错的选择。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 'h', 1: 'e', 2: 'l', 3: 'l', 4: 'o', 5: ' ', 6: 'w', 7: 'o', 8: 'r', 9: 'l', 10: 'd'}\n",
      "<class 'dict'>\n"
     ]
    }
   ],
   "source": [
    "# 用dict() 函数转换成字典\n",
    "s, i = 'hello world', len(s)\n",
    "dict_comp = dict(zip(range(i),s))\n",
    "print(dict_comp)\n",
    "print(type(dict_comp))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 'h', 2: 'e', 3: 'l', 4: 'l', 5: 'o', 6: ' ', 7: 'w', 8: 'o', 9: 'r', 10: 'l', 11: 'd'}\n",
      "<class 'dict'>\n"
     ]
    }
   ],
   "source": [
    "# 对 key 做一点点处理\n",
    "s, i = 'hello world', len(s)\n",
    "dict_comp = {k+1:v for (k,v) in zip(range(i),s)}\n",
    "print(dict_comp)\n",
    "print(type(dict_comp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.4.\t列表生成式实际例子\n",
    "\n",
    "### 5.4.1 指定目录下查找指定后缀的所有文件\n",
    "\n",
    "我们来写一个可以在指定目录下查找指定后缀的所有文件的函数。当然，python 里面有几乎现成的函数，不过我们的函数也很简洁，只有一行。\n",
    "\n",
    "在 list_files_with_ext() 函数中输入目录和后缀，就会返回符合条件的文件名列表。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "def list_files_with_ext(path, ext):\n",
    "    return [file for f in os.listdir(path) if f.endswith(ext)]\n",
    "\n",
    "list_files_with_ext('/Users/david/Pictures/travel', '.jpg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.2 找到指定范围的勾股数\n",
    "\n",
    "一般在列表生成式的结果中用一到两个变量，这里举一个简单的三个变量的例子，用一行列表生成式的程序来生成指定范围内的所有勾股数。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(3, 4, 5),\n",
       " (5, 12, 13),\n",
       " (6, 8, 10),\n",
       " (7, 24, 25),\n",
       " (8, 15, 17),\n",
       " (9, 12, 15),\n",
       " (10, 24, 26),\n",
       " (12, 16, 20),\n",
       " (15, 20, 25),\n",
       " (20, 21, 29)]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(x,y,z) for x in range(1,30) for y in range(x,30) for z in range(y,30) \n",
    " if x**2 + y**2 == z**2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.3 矩阵 flatten 计算\n",
    "\n",
    "将类似 [0,1,2,3,4][5,6,7,8,9] 这样的矩阵铺平成为 [0,1,2,3,4,5,6,7,8,9]\n",
    "\n",
    "这几个例子以及下面的速度比拼可以参考 https://www.analyticsvidhya.com/blog/2016/01/python-tutorial-list-comprehension-examples/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "原始 Matrix: [range(0, 5), range(5, 10), range(10, 15)]\n",
      "使用 for 操作: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]\n",
      "使用列表生成式操作: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]\n"
     ]
    }
   ],
   "source": [
    "# Flatten a Matrix\n",
    "\n",
    "def for_flatten(matrix):\n",
    "    flat = []\n",
    "    for row in matrix:\n",
    "        for x in row:\n",
    "            flat.append(x)\n",
    "    return flat\n",
    "\n",
    "def list_flatten(matrix):\n",
    "    return [x for row in matrix for x in row ]\n",
    "\n",
    "matrix = [ range(0,5), range(5,10), range(10,15)]\n",
    "\n",
    "print('原始 Matrix:',str(matrix))\n",
    "print('使用 for 操作:',str(for_flatten(matrix)))\n",
    "print('使用列表生成式操作:',str(list_flatten(matrix)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4.4 国家首都的匹配\n",
    "\n",
    "实际上这是解决将两个列表如何装配成一个字典的问题，我们通过字典生成式很容易解决这个问题。\n",
    "\n",
    "思路上来说，就是根据一个列表的长度进行循环，然后依次在字典中写入 key 和 value。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "result: {'China': 'Beijing', 'Pakistan': 'Islamabad', 'India': 'New Delhi', 'Nepal': 'Kathmandu', 'Bangladesh': 'Dhaka', 'Bhutan': 'Thimphu'}\n"
     ]
    }
   ],
   "source": [
    "def zip_list(keys, values):\n",
    "    return { keys[i] : values[i] for i in range(len(keys)) }\n",
    "\n",
    "country = ['India', 'Pakistan', 'Nepal', 'Bhutan', 'China', 'Bangladesh']\n",
    "capital = ['New Delhi', 'Islamabad','Kathmandu', 'Thimphu', 'Beijing', 'Dhaka']\n",
    "\n",
    "print('result:', str(zip_list(country, capital)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.5 速度比拼\n",
    "\n",
    "下面的代码可以测试一下用传统的 for 循环、列表生成式和 map 方式完成同样的功能，哪个速度最快 。注意，此段代码需要在 Jupyter Notebook中运行。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100000 loops, best of 3: 6.67 µs per loop\n"
     ]
    }
   ],
   "source": [
    "# for循环函数\n",
    "def for_square(var):\n",
    "    result = []\n",
    "    for i in var:\n",
    "        result.append(i**100)\n",
    "    return result\n",
    "\n",
    "%timeit for_square(range(1,11))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100000 loops, best of 3: 7.06 µs per loop\n"
     ]
    }
   ],
   "source": [
    "# 列表生成式函数\n",
    "def list_square(var):\n",
    "    return [i**100 for i in var]\n",
    "\n",
    "%timeit list_square(range(1,11))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slowest run took 4.38 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
      "1000000 loops, best of 3: 583 ns per loop\n"
     ]
    }
   ],
   "source": [
    "# map函数\n",
    "def map_square(var):\n",
    "    return map(lambda x: x**100, var)\n",
    "\n",
    "%timeit map_square(range(1,11))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "timeit模块提供了测量Python小段代码执行时间的方法。它即可以在命令行界面直接使用，也可以通过导入模块进行调用。\n",
    "\n",
    "我们使用 timeit 的默认参数，选取执行最快的3次的平均值，可以看到，使用 for 和列表生成式相差并不大，用列表生成式略快一些，而使用 map()方式的则要快得多。\n",
    "\n",
    "在下面这里有一篇详细的讨论，也讨论了不同场景下使用不同方法的速度以及原理等：\n",
    "\n",
    "https://stackoverflow.com/questions/1247486/python-list-comprehension-vs-map"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
